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ABSTRACT 

The evaluation of an educational program by portrayal 
of the program rather than the focus of the program is discussed. It 
is suggested that the program evaluator limit his evaluation aims to 
what ha can do and to what the client needs most. It is believed that 
the first duty of the evaluator should be to offer the client a 
comprehensive portrayal of the program. (DB) 
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AN APPROACH TO THE EVALUATION OF INSTRUCTIONAL PROGRAMS* 

Robart E. Staled 

Unlvarsity of Illinois at Urbana-*Chasapalgn 



The evaluator of an Inatructioual prograia is faced with a 
dilenanao Uls design and final report can es^haaise what he can saeasure 
most e£fet!tively giv«i hia laodest resources — or his design and final 
report can eflect the nature of the program, with fidelity to the many 
important perceptions and expectations of it. Both cannot prevallo 



Vihat the evaluator haa to oscy cannot be both a sharp analysis of high** 



lO 



o 
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priority achieveiaent a broad and accurate reflection of the program's 
complex transactions* 

X am saying something t&ore than* "You can't feature both 
product and process in the evaluation study*" 1 am saying, "Any 
focus on the analysis of product ^ process distorts the picture as 
to what the instruction is* Which is ctore important: to tell of sons 

very special things about the program or to provide the most veridical 
portrayal of the program? 

* Paper deUverad at iSSd dnnual Haatiag April 4, 1972, in Gblcago. 
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I an today going to advocato tha lattoi, tbo etsphasis oa 
portrayaii^* at the expc&se of focus* I do not aean this to be sii eaphsr 
sis oa descriptive data rather than judgasental; both are needed la veri- 
dical portrayal evaluations and in sharp-focus evaluations* I do not 
naan this to be an enphasis on sunnatlve rather th(3U formtlve evalua- 
tion; in either case the sane dllenaa appears. I acicnovledge that any 
study that eophasises a particul..^r Issue ,» or a particular decision* or 
a particular goal* at the great dimmitlon of all others* night be a 
aost appropriate research or evaluation effort*— but it should not be 

/'• 

passed off as an evaluation of the prQgrjffi .> 

I 

Our Llaited Yalent s* ' 

’V 

‘'"a; 

We recognise that we are not €»qually able to msaoure all 
outcomes* Sosss coamunity expectations and studest; aspirations s:^e 
deeply hidden. Soaa costs are nore difficult tease out than others* 

Working with a linitad svaluatiou budget we are inclined to confine 
our attention to that which we can aaasure beoi:* 

Many of ua have a great confidence that we can nsasure any 
trait* describe any event* and operationalise any construct though we 
are quick to allow that we cannot do those jobs with a papsr-and-peacll-tast* 
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with a oae-day site visit, or with sn armual of $2,000. In fact, 

oany of the promises ve aalu\ in evaluation pro* osals could not be ful~ 
fll'^ed if we had a £uH**tiiaa evaluation teas? consisting of the ten most 
recent past Presidents of ASRA, at least hrflf of whcm are considered 
experts in evaluatiouo But the point at the nooent is not that evalua** 
cion is a tough Job but that some of our tasks iu any program evaluation 
are tougher than others. And there is a reasonable tendency on our part 
tc feature in the evaluation proposal and period.c reports those tasks 
we do besto 

What Jobs do we do beat? We are incliued to say that the Jobs 
we do best are those our fellow researchers adsr.^ire or least criticise: 
our Iteai enalyseso xoclograss, teak analyses, random a^&mplings, covariance 
a n alyses, attitude scales, mail surveys, and oo on. And. so' we suggest 
to our clients that those things would ba useful to them. We sometimes 
imply that the evaluation would hot be authentic if it did not use some 
of thorn. We fail to realise that many of these procedures were brought 
into our technology as microscopes to examine the minute detail of educa’** 
tlon, not as procaduras for portraying the "whole cloth" of an instruct 
tlonal program. 
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The of Progr<» QomimmtJL 

The whole cloth of an educational program is a grand ac^umular 
tion of intents, transact lone, and out corns « The teachers intend to 
daliv«iC on nany proiaises and to take advantage of many targets of oppor- 
tunltyo Students and parents have their ^rpectatloos and appre h e nsion s o 
Community leaders, social critics, and educationists have "viewed with 
alarm" and "pointed with prides" Each child brings his own complex of 
convictions, sdsunders tendings, and propensities and takes away soma of 
those and still others* Each classroom is a eosimmity, with rules and 
stresses and competition and cos^assiono Yesterday's subgroups are not 
tomorrow’ So Things are learned, unlearned, relearned much as shoelaces 
are knotted, untied, broken, and retied* iuu educational program has 
countless objectives, many of them dormant until a crisis arises. The 
priorities vary over tlm^ from person to person. No statement of program 
objectives ever devised has cons closs to representing the recl-morld 
intents of the people involved in an eUucstional program. 

X should not imply that one cannot get reasonable consensus 
on high-priority objectives for a program. The unspokan objectives— 
safety in the classroom; sharing of work responsibilities; developing a 
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sense of hm»or» a respect for rules 9 sad a tolerance of aaiblgulty; and so 
on» and so on — the unspoken objectives are left to take care of theiaselves^ 
at least until a cri/.ls arises^ And then these objectives may pre eapt 
all otherso One can get simple consensus; and as long as no one takes the 
consensus too seriously^ chiliren can get much more than the primitive 
education that th€i consensus slatemant describes <> 

Consensus Is one of the great slmpllfisrso Theory is another. 
Statistical processes are simplifierao Test scores are simple represents* 
tions of the complexo These sis^llfiers help us by reducing the phenomaaa 
to something within our power of conprehension. But they mislead us by 
saying that education is much less than it really is. We work day by day 
with the siaplificatlfins**ths statements cf obj actives » the central tend* 
ancles 9 thr. criterion tesf4l**sad we \mxfm transfixed by them» losing our 
awareness of the fundsnsntal activitiss of teaching and learning. We do 
It to ourselvea and ve do it to our ai: 4 iwa;eea<. Evaluators should be 
helping people keep in touch with the reality of instruction^ but our 
scrapbooks are full of enlargsmsnts of enlsrgsmtnts. 

Should the evaluator focus on the more prominent features or 
attempt to eid>race the program as a whole? There are many differsnt 
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rittmimHtt q£l thd evAluAtor* Lttt quotft frjm a notice to potententlal 



contractors frost the UoS. Office of Education for evaluation of the 



Ei^erinental Sdbiools Program: 



''Contractors wlXl be required to impleiaent a ccmtpra- 
hensilve evaluation of the project including the follcir- 
ing areas: (1) measurement of the success of the 

overfill project in meeting needs of students and of the 
impact of the project^ (2) saeasureiBent of individual 
project elements; (3) systematic documentation of the 
project; (4) description of the social _ahd political 
forces which lirfliience and shape the ^velopment of 
the project; <S) formative evaluation for the project 
and Its internal evauluation efforts; and C6) cost 
analysis*" 



All these evaluation goals are is|>ortant* But the annual bill for u single 



school's comprehensive evaluation*-'*as deflned-~-would run over a million 
dcllarso That's probably one reason why no evaluation team has ever 
succeeded in doing all those things* In the more ordinary and modest 
situation we should tty to be comprehensive too» but we should limit our 
evaluation aiimi to what .we can do and to what the client needs most* 

What many clients nee d is a credible » thorough representation of what the 



program ist including information about who likes what about it* Clients 
need confirmation of what they know» reminders of things they are over* 
looking, and something in the way of a report to show other people* 

It is difficult for the client to perceive the s<M>pa and 



movemant of the program* The program director' a perspective is partially 



obscured; the outsider's Is evanescento They need to see loore, to share 



■ore la the experience. 1£ the program glows » the evaluation should 
reflect some of ito If the program wobbles » the tremor shovld pass 
through the evaluation reporto The first duty of the evaluator should be 
to offer the client a comprehensive portrayal of the program^ 

The client may want something elsce OoK. He laay want more 
than portrayals He may want something other than portrayalo O.Ko He 
may want a concentrated examination of the pursuit of a few objectives* 

He may want a study of the causes of success or failure* or a study of 
transportability* or a study of the efficiency of the programo If he 
has the resources* ha should get what he wants » But he should not be 
encouraged to pursue those costly and elusive phantoms if what he needs 
is a substantive portrayal of his Instructional program* 

It's u toug^ choice: focus or portrayal* The evalurtor has 

to help figure out which will be more useful* Many of us are biased in 
favor of focuso 

Think for a aoaent what a book review has become in the 
Sunday Times : an opportunity for the reviewer to get something off bla 
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chest, a chsaca to psaipiir a pat idee, with at laost a tenuous oonnectiou 
to the book re'viewad* kern pcogm ewsluatioa studies connected to 
prograas by more then a tenuous shoestring? Are they little nore than 
the exploiting of an instructional researcher* s hunch or a psychoaetri- 
dan* 8 faadnation? Vo owe the people acre than that« 
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